SYSTEM AND METHOD FOR INPUTTING, RETRIEVING, 
ORGANIZING AND ANALYZING DATA 

FIELD OF INVENTION 

5 The present invention relates to data collection systems and more particularly 

pertains to a system for inputting, retrieving, and analyzing data in an organized and 
thus effective manner. 

M BACKGROUND OF INVENTION 

% 10 The collection and analysis of data is an essential component of making various 

tea* 

P decisions in the marketplace. The collection of data may take on many forms 

ry including, but not limited to recording video, images, audio, hard numerical data, or 

H * the like. While the information present in data is often invaluable in many respects, 

jpC'::> 

s it can somtetimes be difficult to extract that information on which a decision is to be 

L 15 based. For example, in cases where the amount of collected data rises to substantial 
H levels, confusion may arise if the data is not maintained in order In addition, a lack 

O of organization may inhibit an extent to which a team can analyze the data 

effectively. 

20 Moreover, even if a proper conclusion has been arrived at, there may be a 

subsequent need to review the basis on which the conclusion was founded for one 
reason or another. Here again, lack of organization often obscures the train of 
thought that rationalized the proper conclusion. As such, there is a substantial need 
for a system and method for inputting, retrieving, and analyzing data in an organized 

25 and thus effective manner. 

Further research includes the following pieces of literature which are each 
incorporated herein by reference: 

30 LArai, T., Aust, D., and Hudson, S. E. PaperLink: A Technique for Hyperlinking 
from Real Paper to Electronic Content, in Proceedings of CHI 9 97 (Atlanta GA, 
March 1997), ACM Press, 327-334. 



2. Bellotti, V., Integrating Theoreticians' and Practioners' Perspectives with Design 
Rationale, in Proceedings oflnterCHI '93 (Amsterdam, April 1993), ACM Press, 
101-106. 

3. Carroll, J., Alport, S., Karat, J., Van Dusen, M., and Rosson, M. Raison d'Etre: 
Capturing design history and rationale in multimedia narratives, in Proceedings of 
CHI '94 (Boston MA, April 1994), ACM Press, 192-197. 

4. Conklin, J. and Begeman, M.L. gIBIS: A Hypertext Tool for Exploratory Policy 
Discussion, in Proceedings ofCSCW '88 (Portland OR, September 1988), ACM 
Press, 140-152. 

5. Holtzblatt, K. and Jones, S., Conducting and Analyzing a Contextual Interview, in 
Namioka and Schuler (Eds.) Participatory Design: Principles and Practice,. 
Lawrence Erlbaum Associates, Hillsdale, NJ, 1993. 

6. Hughes, J., King, V., Rodden, T., and Andersen, H. The Role of Ethnography in 
Interactive Systems Design. Interactions, 2, 2 (April 1995), 57-65. 

7. Jordan, B., and Henderson, A. Interaction Analysis: Foundations and Practice. The 
Journal of the Learning Sciences 4, 1 (1995), 39-103. 

8. Landay, J. A., and Myers, B. A. Interactive Sketching for the Early Stages of User 
Interface Design, in Proceedings of CHI '95 (Denver CO, May 1995), ACM Press, 
43-50. 

9. Lewis, S., Mateas, M., Palmiter, S., and Lynch, G. Ethnographic Data for Product 
Development: A Collaborative Process. Interactions, 3, 6 (November/December 
1996), 53-69. 



lO.Mackay, W.E., and Pagani, D.S. Video Mosaic: Laying Out Time in a Physical 
Space, in Proceedings of Multimedia '94 (San Francisco, CA, October 1994), 
ACM Press, 165-172. 

1 LMoran, T. P. and Anderson, R. J., The Workaday World as a Paradigm for 
CSCW Design, in Proceedings ofCSCW '90 (Los Angeles CA, October 1990), 
ACM Press, 381-393. 

12. Moran, T. P., Chiu, P., Harrison, S., Kurtenbach, G., Minneman, S., and van 
Melle, W. Evolutionary Engagement in an Ongoing Collaborative Work Process: A 
Case Study, in Proceedings of CSCW '96 (Cambridge MA, November 1996), 
ACM Press, 150-159. 

13. Moran, T. P., Palen, L., Harrison, S., Chiu, P., Kimber, D., Minneman, S., van 
Melle, W., and Zellweger, P. "I'll Get That Off the Audio": A Case Study of 
Salvaging Multimedia Meeting Records, in Proceedings of CHI '97 (Atlanta GA, 
March 1997), ACM Press, 202-209. 

14.0'Hara, K., and Sellen, A. A Comparison of Reading Paper and On-Line 
Documents, in Proceedings of CHI '97 (Atlanta GA, March 1997), ACM Press, 
335-342. 

15.Pedersen, E., McCall, K., Moran, T., and Halasz, F. Tivoli: an Electronic 
Whiteboard for Informal Workgroup Meetings, in Proceedings oflnterCHI '93 
(Amsterdam, May 1993), ACM Press, 391-398. 

16.Sellen, A., and Harper, R. Paper as an Analytic Resource for the Design of New 
Technologies, in Proceedings of CHI '97 (Atlanta GA, March 1997), ACM Press, 
319-326. 

17.Suchman, L. Representations of Work Communications of the ACM 38, 9 
(entire issue). 
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18.Suchman, L. A., and Trigg, R. H. Understanding Practice: Video as a Medium 
for Reflection and Design, in Greenbaum and Kyng (Eds.) Design at Work: 
Cooperative Design of Computer Systems. Lawrence Erlbaum Associates, 
5 Hillsdale, NJ, 1993. 



19.Swaminathan, K., and Sato, S. Interaction Design for Large Displays. 
Interactions 4, l(Jan-Feb. 1997), 15-24. 

10 20.Wall, P., and Mosher, A. Representations of Work: Bringing Designers and Users 
Together, in Proceedings ofPDC '94 (Chapel Hill NC, October 1994), ACM 
Press, 87-98. 

21.Wellner, P. Interacting with Paper on the DigitalDesk. Communications of the 
15 ACM 36, 7 (1993), 97-96. 

SUMMARY OF INVENTION 

The present invention overcomes the problems associated with data collection, retrieval, 
and analysis by including at least one remote input device for recording remote data and 
20 a host computer for receiving and storing the remote data from the remote input device. 
Also included is at least one local input device coupled to the host computer for 
retrieving the remote data within the host computer. Associated therewith is at least one 
local output device that is coupled to the host computer for retrieving the data. 

25 BRIEF DESCRIPTION OF DRAWINGS 

The invention will be better understood when consideration is given to the following 
detailed description thereof. Such description makes reference to the annexed 
drawings wherein: 



30 



Figure 1 is a perspective view of the present invention; 
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Figure 2 is a schematic diagram of the present invention; 
Figure 3 is a flowchart delineating the method of use of the present invention; 
5 Figure 4 is a front view of one of the lists of the present invention; 
Figure 5 is a front view of one of the cards of the present invention; 
Figure 6 is a front view of one of the sleeves of the present invention; 

5 10 

3 Figure 7 is a rear view of one of the sleeves of the present invention; 

CP Figure 8 diagrams the elements of the Analysis Framework which are incorporated 

1 into the present invention; 

f* 15 

f? Figure 9 illustrates an example for each element of the Analysis Framework of 

Figure 8; 



20 



Figure 10 depicts the Multimedia Analysis Software Tool (MAST) which supports 
data capture and storage, data entry, and searching and display capabilities and 
further manages the interactions between the objects, devices, equipment, and 
displays of the present invention; 



25 



Figure 11 is a front view of the displays of the present invention; 
Figure 12 illustrates an example of a window of the present invention; 
30 Figure 13 illustrates yet another example of a window of the present invention; 



Figure 14 is an illustration of the use of walls as display spaces for information, in 
accordance with one embodiment of the present invention; and 

Figure 15 illustrates the three different types of commands associated with the 
Barcode Command Slate of the present invention. 

DISCLOSURE OF INVENTION 

Hardware Overview 

With reference to Figure 1, the present invention includes at least one remote input 
device 10, a host computer 12, at least one local input device 14, and at least one local 
output device 16. Generally speaking, the remote input device 10 serves for recording 
remote data. In one embodiment, a plurality of remote input devices 10 are included for 
accomplishing this task. For example, the remote input devices 10 may include a video 
camera 18 for recording video signals, an image camera 20 for recording image signals, 
an audio recorder 22 for recording audio signals, and a data recorder 24 for recording 
hard data. It should be noted that any type of sensor, hard-line or wireless keyboard 26, 
measuring device, or input mechanism may be included for accumulating the remote 
data. 

One task of the host computer 12 is to receive and store the remote data from the remote 
input devices 10. This may be accomplished by any type of parallel, serial, optical, radio 
transceiver, or any other device or connection capable of communicating the remote data 
between the remote input devices 10 and the host computer 12. In one embodiment, the 
host computer 12 may take the form of a media server, workstation, or any other type of 
computer capable of carrying out the necessary tasks. The manner in which the host 
computer 12 controls the present invention will be set forth hereinafter. 

The local input device 14 is coupled to the host computer 12 for retrieving the remote 
data within the host computer 12. This may be accomplished within a lab or any other 
environment in which a single person or a group of people are capable of interacting 
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while the remote data is being retrieved. In one embodiment, a plurality of local input 
devices 14 is included. For example, the local input devices 14 may include a keyboard 
26, a bar code scanner 28, or any other input device capable of retrieving the remote data 
within the host computer 12. While one task of the local input devices 14 include 
5 retrieving the remote data within the host computer 12, yet another task associated 
therewith includes inputting local data into the host computer 12. To accomplish this 
additional task, the local input device 14 includes an electronic drawing board 29, the 
keyboard 26, or any other device for inputting local data. 

O 10 With respect to the local output device 16, the same is coupled to the host computer 12 

%i for exhibiting the data. In various embodiments, the local output device 16 may include 

% a printer 30, a display 32, an audio mechanism 34, or any other type of output 

mechanism. Accordingly, output may be in the form of a hard copy or a visual image(s). 

\* In one embodiment, a pair of displays 32 may be provided including a projection screen 

p • 15 38 and a computer monitor 40. 
Si 

flj 

2 In use, the host computer 12 operates to effectively input, retrieve, and organize the 

remote data so that additional local data may be generated and "linked" to the remote 
data in a lab environment amongst a number of people. This operation is delineated in 
20 the flowchart of Figure 3 and may be carried out by the host computer 12 using 
specifically tailored software. 

Software Overview 

25 A preferred embodiment is written using JAVA, C, and the C++ language and 

utilizes object oriented programming methodology. Object oriented programming 
(OOP) has become increasingly used to develop complex applications. As OOP 
moves toward the mainstream of software design and development, various software 
solutions require adaptation to make use of the benefits of OOP. A need exists for 

30 these principles of OOP to be applied to a messaging interface of an electronic 
messaging system such that a set of OOP classes and objects for the messaging 
interface can be provided. 
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OOP is a process of developing computer software using objects, including the steps 
of analyzing the problem, designing the system, and constructing the program. An 
object is a software package that contains both data and a collection of related 
structures and procedures. Since it contains both data and a collection of structures 
5 and procedures, it can be visualized as a self-sufficient component that does not 
require other additional structures, procedures or data to perform its specific task. 
OOP, therefore, views a computer program as a collection of largely autonomous 
components, called objects, each of which is responsible for a specific task. This 

H' concept of packaging data, structures, and procedures together in one component or 

O 10 module is called encapsulation. 

W In general, OOP components are reusable software modules which present an 

£t interface that conforms to an object model and which are retrieved at run-time 

through a component integration architecture. A component integration architecture 
U 15 is a set of architecture mechanisms which allow software modules in different 
y process spaces to utilize each others capabilities or functions. This is generally done 

O by assuming a common component object model on which to build the architecture. 

It is worthwhile to differentiate between an object and a class of objects at this point. 

An object is a single instance of the class of objects, which is often just called a 
20 class. A class of objects can be viewed as a blueprint, from which many objects can 

be formed. 

OOP allows the programmer to create an object that is a part of another object. For 
example, the object representing a piston engine is said to have a composition- 
25 relationship with the object representing a piston. In reality, a piston engine 

comprises a piston, valves and many other components; the fact that a piston is an 
element of a piston engine can be logically and semantically represented in OOP by 
two objects. 

30 OOP also allows creation of an object that "depends from" another object. If there 
are two objects, one representing a piston engine and the other representing a piston 
engine wherein the piston is made of ceramic, then the relationship between the two 



objects is not that of composition. A ceramic piston engine does not make up a 
piston engine. Rather it is merely one kind of piston engine that has one more 
limitation than the piston engine; its piston is made of ceramic. In this case, the 
object representing the ceramic piston engine is called a derived object, and it 
inherits all of the aspects of the object representing the piston engine and adds 
further limitation or detail to it. The object representing the ceramic piston engine 
"depends from" the object representing the piston engine. The relationship between 
these objects is called inheritance. 

When the object or class representing the ceramic piston engine inherits all of the 
aspects of the objects representing the piston engine, it inherits the thermal 
characteristics of a standard piston defined in the piston engine class. However, the 
ceramic piston engine object overrides these ceramic specific thermal characteristics, 
which are typically different from those associated with a metal piston. It skips over 
the original and uses new functions related to ceramic pistons. Different kinds of 
piston engines have different characteristics, but may have the same underlying 
functions associated with it (e.g., how many pistons in the engine, ignition 
sequences, lubrication, etc.). To access each of these functions in any piston engine 
object, a programmer would call the same functions with the same names, but each 
type of piston engine may have different/overriding implementations of functions 
behind the same name. This ability to hide different implementations of a function 
behind the same name is called polymorphism and it greatly simplifies 
communication among objects. 

With the concepts of composition-relationship, encapsulation, inheritance and 
polymorphism, an object can represent just about anything in the real world. In fact, 
the logical perception of the reality is the only limit on determining the kinds of 
things that can become objects in object-oriented software. Some typical categories 
are as follows: 

• Objects can represent physical objects, such as automobiles in a traffic-flow 
simulation, electrical components in a circuit-design program, countries in an 
economics model, or aircraft in an air-traffic-control system. 



- 10- 

• Objects can represent elements of the computer-user environment such as 
windows, menus or graphics objects. 

• An object can represent an inventory, such as a personnel file or a table of the 
latitudes and longitudes of cities. 

5 • An object can represent user-defined data types such as time, angles, and 
complex numbers, or points on the plane. 

With this enormous capability of an object to represent just about any logically 
"1 separable matters, OOP allows the software developer to design and implement a 

O 10 computer program that is a model of some aspects of reality, whether that reality is a 
\i physical entity, a process, a system, or a composition of matter. Since the object can 

2= represent anything, the software developer can create an object which can be used as 

H a component in a larger software project in the future. 

si 

^ 15 If 90% of a new OOP software program consists of proven, existing components 

D 

sj made from preexisting reusable objects, then only the remaining 10% of the new 

software project has to be written and tested from scratch. Since 90% already came 
from an inventory of extensively tested reusable objects, the potential domain from 
which an error could originate is 10% of the program. As a result, OOP enables 
20 software developers to build objects out of other, previously built objects. 

This process closely resembles complex machinery being built out of assemblies and 
sub-assemblies. OOP technology, therefore, makes software engineering more like 
hardware engineering in that software is built from existing components, which are 
25 available to the developer as objects. All this adds up to an improved quality of the 
software as well as an increased speed of its development. 

Programming languages are beginning to fully support the OOP principles, such as 
encapsulation, inheritance, polymorphism, and composition-relationship. With the 
30 advent of the C++ language, many commercial software developers have embraced 
OOP. C++ is an OOP language that offers a fast, machine-executable code. 



Furthermore, C++ is suitable for both commercial-application and systems- 
programming projects. For now, C++ appears to be the most popular choice among 
many OOP programmers, but there is a host of other OOP languages, such as 
Smalltalk, Common Lisp Object System (CLOS), and Eiffel. Additionally, OOP 
capabilities are being added to more traditional popular computer programming 
languages such as Pascal. 

The benefits of object classes can be summarized, as follows: 

• Objects and their corresponding classes break down complex programming 
problems into many smaller, simpler problems. 

• Encapsulation enforces data abstraction through the organization of data into 
small, independent objects that can communicate with each other. 
Encapsulation protects the data in an object from accidental damage, but 
allows other objects to interact with that data by calling the object's member 
functions and structures. 

• Subclassing and inheritance make it possible to extend and modify objects 
through deriving new kinds of objects from the standard classes available in 
the system. Thus, new capabilities are created without having to start from 
scratch. 

• Polymorphism and multiple inheritance make it possible for different 
programmers to mix and match characteristics of many different classes and 
create specialized objects that can still work with related objects in 
predictable ways. 

• Class hierarchies and containment hierarchies provide a flexible mechanism 
for modeling real-world objects and the relationships among them. 

• Libraries of reusable classes are useful in many situations, but they also have 
some limitations. For example: 

• Complexity. In a complex system, the class hierarchies for related classes 
can become extremely confusing, with many dozens or even hundreds of 
classes. 
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• Flow of control. A program written with the aid of class libraries is still 
responsible for the flow of control (i.e., it must control the interactions 
among all the objects created from a particular library). The programmer has 
to decide which functions to call at what times for which kinds of objects. 
5 • Duplication of effort. Although class libraries allow programmers to use and 
reuse many small pieces of code, each programmer puts those pieces together 
in a different way. Two different programmers can use the same set of class 
libraries to write two programs that do exactly the same thing but whose 
internal structure (i.e., design) may be quite different, depending on hundreds 
10 of small decisions each programmer makes along the way. Inevitably, 

similar pieces of code end up doing similar things in slightly different ways 
and do not work as well together as they should. 

Class libraries are very flexible. As programs grow more complex, more 
15 programmers are forced to reinvent basic solutions to basic problems over and over 
again. A relatively new extension of the class library concept is to have a framework 
of class libraries. This framework is more complex and consists of significant 
collections of collaborating classes that capture both the small scale patterns and 
major mechanisms that implement the common requirements and design in a 
20 specific application domain. They were first developed to free application 

programmers from the chores involved in displaying menus, windows, dialog boxes, 
and other standard user interface elements for personal computers. 

Frameworks also represent a change in the way programmers think about the 
25 interaction between the code they write and code written by others. In the early days 
of procedural programming, the programmer called libraries provided by the 
operating system to perform certain tasks, but basically the program executed down 
the page from start to finish, and the programmer was solely responsible for the flow 
of control. This was appropriate for printing out paychecks, calculating a 
30 mathematical table, or solving other problems with a program that executed in just 
one way. 
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The development of graphical user interfaces began to turn this procedural 
programming arrangement inside out. These interfaces allow the user, rather than 
program logic, to drive the program and decide when certain actions should be 
performed. Today, most personal computer software accomplishes this by means of 
5 an event loop which monitors the mouse, keyboard, and other sources of external 
events and calls the appropriate parts of the programmer's code according to actions 
that the user performs. The programmer no longer determines the order in which 
events occur. Instead, a program is divided into separate pieces that are called at 
unpredictable times and in an unpredictable order. By relinquishing control in this 
10 way to users, the developer creates a program that is much easier to use. 

Nevertheless, individual pieces of the program written by the developer still call 
libraries provided by the operating system to accomplish certain tasks, and the 
programmer must still determine the flow of control within each piece after it's 
called by the event loop. Application code still "sits on top of the system. 

15 

Even event loop programs require programmers to write a lot of code that should not 
need to be written separately for every application. The concept of an application 
framework carries the event loop concept further. Instead of dealing with all the 
nuts and bolts of constructing basic menus, windows, and dialog boxes and then 
20 making these things all work together, programmers using application frameworks 
start with working application code and basic user interface elements in place. 
Subsequently, they build from there by replacing some of the generic capabilities of 
the framework with the specific capabilities of the intended application. 

25 Application frameworks reduce the total amount of code that a programmer has to 
write from scratch. However, because the framework is really a generic application 
that displays windows, supports copy and paste, and so on, the programmer can also 
relinquish control to a greater degree than event loop programs permit. The 
framework code takes care of almost all event handling and flow of control, and the 

30 programmer's code is called only when the framework needs it (e.g., to create or 
manipulate a proprietary data structure). 
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A programmer writing a framework program not only relinquishes control to the 
user (as is also true for event loop programs), but also relinquishes the detailed flow 
of control within the program to the framework. This approach allows the creation 
of more complex systems that work together in interesting ways, as opposed to 
5 isolated programs, having custom code, being created over and over again for similar 
problems. 

Thus, as is explained above, a framework basically is a collection of cooperating 
classes that make up a reusable design solution for a given problem domain. It 
10 typically includes objects that provide default behavior (e.g., for menus and 

windows), and programmers use it by inheriting some of that default behavior and 
overriding other behavior so that the framework calls application code at the 
appropriate times. 



I= 15 There are three main differences between frameworks and class libraries: 

• Behavior versus protocol. Class libraries are essentially collections of 
behaviors that you can call when you want those individual behaviors in the 
program. A framework, on the other hand, provides not only behavior but 
also the protocol or set of rules that govern the ways in which behaviors can 

20 be combined, including rules for what a programmer is supposed to provide 

versus what the framework provides. 

• Call versus override. With a class library, the code the programmer 
instantiates objects and calls their member functions. It's possible to 
instantiate and call objects in the same way with a framework (i.e., to treat 

25 the framework as a class library), but to take full advantage of a framework's 

reusable design, a programmer typically writes code that overrides and is 
called by the framework. The framework manages the flow of control among 
its objects. Writing a program involves dividing responsibilities among the 
various pieces of software that are called by the framework rather than 
30 specifying how the different pieces should work together. 
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• Implementation versus design. With class libraries, programmers reuse only 
implementations, whereas with frameworks, they reuse design. A framework 
embodies the way a family of related programs or pieces of software work. It 
represents a generic design solution that can be adapted to a variety of 
5 specific problems in a given domain. For example, a single framework can 

embody the way a user interface works, even though two different user 
interfaces created with the same framework might solve quite different 
interface problems. 

O 10 Thus, through the development of frameworks for solutions to various problems and 
H programming tasks, significant reductions in the design and development effort for 

\. software can be achieved. A preferred embodiment of the invention utilizes 

5 HyperText Markup Language (HTML) to implement documents on the Internet 

together with a general-purpose secure communication protocol for a transport 
¥* 15 medium between the client and the Newco. HTTP or other protocols could be readily 
q substituted for HTML without undue experimentation. Information on these 

products is available in T. Berners-Lee, D. Connoly, "RFC 1866: Hypertext Markup 
P Language - 2.0" (Nov. 1995); and R. Fielding, H, Frystyk, T. Berners-Lee, J . Gettys 

and J.C. Mogul, "Hypertext Transfer Protocol - HTTP/1.1: HTTP Working Group 
20 Internet Draft" (May 2, 1996). HTML is a simple data format used to create 
hypertext documents that are portable from one platform to another. HTML 
documents are SGML documents with generic semantics that are appropriate for 
representing information from a wide range of domains. HTML has been in use by 
the World-Wide Web global information initiative since 1990. HTML is an 
25 application of ISO Standard 8879; 1986 Information Processing Text and Office 
Systems; Standard Generalized Markup Language (SGML). 

To date, Web development tools have been limited in their ability to create dynamic 
Web applications which span from client to server and interoperate with existing 
30 computing resources. Until recently, HTML has been the dominant technology used 
in development of Web-based solutions. However, HTML has proven to be 
inadequate in the following areas: 
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• Poor performance; 

• Restricted user interface capabilities; 

• Can only produce static Web pages; 

• Lack of interoperability with existing applications and data; and 

• Inability to scale. 

Sun Microsystem's Java language solves many of the client-side problems by: 

• Improving performance on the client side; 

• Enabling the creation of dynamic, real-time Web applications; and 

• Providing the ability to create a wide variety of user interface components. 

With Java, developers can create robust User Interface (UT) components. Custom 
"widgets" (e.g., real-time stock tickers, animated icons, etc.) can be created, and 
client-side performance is improved. Unlike HTML, Java supports the notion of 
client-side validation, offloading appropriate processing onto the client for improved 
performance. Dynamic, real-time Web pages can be created. Using the above- 
mentioned custom UI components, dynamic Web pages can also be created. 

Sun's Java language has emerged as an industry-recognized language for 
"programming the Internet." Sun defines Java as: "a simple, object-oriented, 
distributed, interpreted, robust, secure, architecture-neutral, portable, high- 
performance, multithreaded, dynamic, buzzword-compliant, general-purpose 
programming language. Java supports programming for the Internet in the form of 
platform-independent Java applets." Java applets are small, specialized applications 
that comply with Sun's Java Application Programming Interface (API) allowing 
developers to add "interactive content" to Web documents (e.g., simple animations, 
page adornments, basic games, etc.). Applets execute within a Java-compatible 
browser (e.g., Netscape Navigator) by copying code from the server to client. From 
a language standpoint, Java's core feature set is based on C++. Sun's Java literature 
states that Java is basically, "C++ with extensions from Objective C for more 
dynamic method resolution." 
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Another technology that provides similar function to JAVA is provided by Microsoft 
and ActiveX Technologies, to give developers and Web designers wherewithal to 
build dynamic content for the Internet and personal computers. ActiveX includes 
5 tools for developing animation, 3-D virtual reality, video and other multimedia 

content. The tools use Internet standards, work on multiple platforms, and are being 
supported by over 100 companies. The group's building blocks are called ActiveX 
Controls, small, fast components that enable developers to embed parts of software 
in hypertext markup language (HTML) pages. ActiveX Controls work with a variety 
p 10 of programming languages including Microsoft Visual C++, Borland Delphi , 
% Microsoft Visual Basic programming system and, in the future, Microsoft's 

development tool for Java, code named " Jakarta." ActiveX Technologies also 

ft! 

jp includes ActiveX Server Framework, allowing developers to create server 

applications. One of ordinary skill in the art readily recognizes that ActiveX could 
M 15 be substituted for JAVA without undue experimentation to practice the invention. 

;?"'":< 

jj Description of Operation 

In use, remote data is first recorded by way of the remote input devices 10 and stored in 
20 the host computer 12 in a digital format, as shown in box 50 of Figure 3. Thereafter, the 
host computer 12 serves to identify the remote data by way of one of the local input 
devices 14 such as the keyboard 26. For example, once the remote data is stored in a 
digital format, an identifier may be entered which includes a written description 44 and 
an identification code 42. See box 52 of Figure 3. In one embodiment, the written 
25 description 44 includes both a title and a short summary. It should be noted that the 
remote data may also be categorized by type, i.e. video, image, audio, hard data, etc., or 
by any other standard. 

With all of the remote data loaded and identified, the same may be browsed by using 
30 the keyboard 26 and the display 32 of the host computer 12. In the alternative, the 
host computer 12 may be used to outpu ts lists 46 of the identifiers by way of one of 
the local output devices 16 such as the printer 30 or display 32, as shown in box 54 
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of Figure 3. An example of such lists is shown in Figure 4. The content of such lists 
46 may be regulated by the users by selecting certain criteria regarding the type of 
remote data or items within the associated written descriptions 44. Given these 



output lists 46, the users mav^perusejhe remote data. For reasons that will become 
apparent hereinafter, the identification codes 42 of the identifiers are printed adjacent 
to the associated written description 44. In one embodiment, the identification code 
42 takes the form of a bar code. 

The bar code comprises alternating black and white spaces. The information is 
contained in the relative length of the runs of the black. Each code begins with a 
special marker of a black space followed by a white space followed by another black 
space, where the length of the white space is approximately equal to the length of 
each of the surrounding black spaces. Subsequent runs of black are compared with 
the average length of the runs in the start sequence and are interpreted as multiples 
of that length. An ending sequence similar to the start sequence is required for a 
valid code. 

While browsing the printed lists 46 of remote data, the host computer 12 is further 
capable of outputting t he remote data in(reStm^)>y way of the appropriate output 
device, i.e. display 32, upon one of the identifiers being selected by the user. See box 56 
of Figure 3. To select a desired identifier from the list, the user may simply utilize the 
bar code scanner 28 to provide the host computer 12 with the identifier. Upon receipt of 
the identifier, the host computer 12 allows immediate output of the corresponding 
remote data to be experienced by the users. 

As the remote data is being analyzed in real-time via the foregoing method, the users 
may wish to record local data, i.e. discussion, comments, etc., regarding certain groups 
of the remote data. See box 58 of Figure 3. Such local data may include information 
relating to patterns or trends identified in the remote data. To acco mmodate t his need of 
recording local data, the host computer 12 allows storage of the local data entered by 
way of one of the local input devices 14 suc h as the keyboard 26. Actual entry of the 
local data may take place in a "window" on the display 32 after which the corresponding 
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remote data may be selected so that the local data may be "linked" to the remote data. 
Note box 60 of Figure 3. In the present description, the term "linked" refers to one piece 
of data being associated with another. In one embodiment, these may even refer to one 
piece of data being listed whenever another "linked" piece of data is retrieved. In 
5 another embodiment, "linked" may refer to various pieces of data being stored in a 
similar file. 

After the generation of a multitude of local data linked to various remote data, the users 
may wish to discuss various patterns and make conclusions in a physical environment. 

10 In these cases, the printer 30 and the electronic drawing board 29 may be employed. 

First, the host computer 12 outputs the local data in the form of a print out by way of the 
printer 30. While any printer 30 may be used, in one embodiment, the printer 30 is a 
card printer which is capable of printing small, "index card-sized" cards 48. On these 
cards 48, not only is the local data printed, but also the remote data linked to the local 

15 data along with a corresponding identification code 42. As such, the remote data 

"linked" to the analysis may be viewed in real-time by way of the bar code reader in a 
manner similar to that associated with the aforementioned lists 46. It should be noted 
that the cards 48 may also be printed out with just an identification code 42 
representative of a group of "linked" remote data if so desired. 

20 

Once a multitude of the cards 48 are generated, the same may be situated on the 
electronic drawing board 29. In the present description, the electronic drawing board 29 
may include any medium wherein a writing utensil may be used to create a visible image 
and such image may be stored on the host computer 12. Once the image is stored and no 

25 longer needed, the visible image may be erased and the board reused. In one 

embodiment, the electronic drawing board 29 includes a conventional marker and a 
white drawing board 29 that functions in a manner similar to a "glider" mouse common 
on many conventional computer keyboards 26. When the local data generated on the 
drawing board 29 is stored, the same may be "linked" to the local data of the cards 48. 

30 Of course, the drawing board 29 and printer 30 may be used independently depending on 
the needs of the users. 
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In order that the cards 48 may be easily attached, removed and maneuvered on the 
electronic drawing board 29, a plurality of sleeves 70 is provided, as shown in Figures 6 
and 7. Such sleeves 70 may be constructed from a transparent plastic for housing one of 
the cards 48 while further allowing the same to be viewed. Finally, magnets 72 may be 
5 situated on a rear face of the sleeves 70 for allowing the cards 48 to be removably 
attached to the drawing board 29 in a convenient manner. 



Example of Use 

10 One example of the manner of use of the present invention will now be set forth. In the 
present example, the present invention may be used to determine whether an automobile 
related product works well in its intended environment. First, the remote input devices 
10 may be used to collect still images and videoof the product in use by a number of 
r persons. The remote input devices 10 may also be employed to record numerous audible 

H* 15 interviews with the persons relating to their experience with the product. Upon all of the 
Sj remote data being collected, the same is consolidated at a central lab where it is stored 

into the host computer 12 in a computer readable format, namely a digital format^ As 
each piece of remote data is stored, a written description 44 and id^tificatijp!nXQda42, 
or identifier, is stored therewith. 

20 

With the remote data stored and identified, a team of users may begin perusing the 
remote data via the various local output devices 16. To facilitate such process, print 
outs may be generated by the host computer 12 that list each of the identifiers of the 
remote data. With these lists 46, the team may use the bar code scanner 28 to 
25 immediately retrieve the remote data and experience the same in real-time. This may be 
accomplished by merely scanning the listed identification code 42 of the identifier with 
the bar code scanner 28. 



As the team begins noticing patterns, trends or anything else noticeable about certain 
30 pieces of remote data, further input may be entered in the form of lo cal d ata. Using the 
keyboard 26, the team may also link the local data to the remote data on which it is 
based. Thereafter, the various local data is printed by the host computer 12 on index 
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cards 48 which indicate not only the local data, but also the identifiers of the linked 
remote data. By this feature, the remote data may be retrieved at any time on the index 
card 48 through use of the bar code scanner 28. As an option, the users may simply 
forego inputting local data on the cards 48 and simply list groups of remote data and 
associated identifiers. Further, in the alternative, the cards 48 may not be generated at 
all. 

To further utilize the remote data and any local data generated thus far for the purpose of 
coming to any additional conclusions, the team may utilize the electronic drawing board 
29. In use, the cards 48 may be movably situated on the drawing board 29 after which 
the marker may be used to delineate various comments and discussions with respect to 
the cards 48 or any other information. The local data which is generated with the 
drawing board 29 may optionally be "linked" to any corresponding local data generated 
by the keyboard 26 and further stored in the host computer 12. 

With the present system, the process of analyzing data is well documented and 
organized from start to finish to not only aid the analysis and decision making process, 
but also provide an easily accessible historical account of how such decisions were 
made. 



-22- 



BEST MODES FOR CARRYING OUT THE INVENTION 



Introduction 

5 



The purpose of the present invention is to improve how teams create requirements 
documents by supporting the process of transforming data into insights. In the spirit 
of Video Mosaic a goal is to support an iterative process that relies on sketching 

10 ideas, viewing and displaying a lot of material at one time, and utilizing the spatial 
relationships between materials to provide meaning for the work being done. It is 
important to enable a team to manipulate, record, and review its work, in any format 
used, throughout the entire design process. Enabling a team to effectively create 
requirements documents is based on three separate, but interdependent elements that 

15 are highlighted hereinafter: 



1 . Analysis Framework 



An analysis framework and a structured process are necessary to facilitate the 
20 analysis of enormous amounts of qualitative data. Without an agreed-upon process, 
the team spends valuable time negotiating a process; and lack of an analysis 
framework can lead to ad-hoc and possibly incomplete analysis. 



2. Managing and Displaying the Data 

25 

Software tools to digitize, store, index, and access large amounts of multimedia data 
enable a team to easily work with any of the data by following established links. 



3. Connecting Paper to the Computer 



30 
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It is important to create a working environment that facilitates team interaction as 
well as the capture of design rationales through the innovative, integrated use of 
paper, whiteboards, and computers. 

5 The present invention is an immersive team environment. It consists of software 
tools to manage multimedia data, numerous display devices and writing surfaces, 
and the use of barcodes to link paper, displays, and data. The present discussion 
presents the key features of the present invention: the Analysis Framework, 
:u Managing the Data, Displaying the Data, and Connecting Paper to the Computer. 

10 Examples of use are presented throughout the present description. 

•BBS' 

o 

; :;.J 

f\l Background 

CP 

The experiences transforming data into requirements documents led to the creation 
L ; 15 of the present invention. Over the past few years, much participation has been made 
O in projects to develop knowledge management systems, collaborative tools for multi- 

p site project teams, mobile computing applications, and interactive multimedia 

applications. For example, an extensive project to develop a knowledge management 
vision for an organization has been completed. The resultant documents - a Vision 
20 Report, and the supporting Human-Centered Design Analysis Report - were based 
on data collected at 17 sites in seven countries, where a team of four people 
observed and interviewed more than 150 people over a three month period. 

During the knowledge management project, a simple, custom-built data structuring 
25 software application was used to catalogue the data and begin the analysis. However, 
the in-depth analysis was done without the benefit of computerized tools. Using 
well-known knowledge management frameworks to structure the analysis, an 
electronic whiteboard was used to sketch scenarios and Postlts™ placed on a 
whiteboard to create diagrams, matrices, and tables. The Postlts and electronic 
30 whiteboard printouts helped to display key issues and the problems to address. While 
this process worked, it was cumbersome, inefficient and time-consuming. In the end 
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a scattered pile of Postlts were left and much of the project history was not 
preserved. 

During the knowledge management project, it was documented that all of the 
activities by recording the work as digital images and 35mm slides. Using these 
materials, it was realized the key factors that influenced the way design sessions 
were conducted. The outcome was a set of goals for the present invention: to support 
team-based analysis, facilitate informal interaction, utilize technology to improve the 
process, and enable teams to use common objects including pen and paper. 

A basic premise of the present invention design is that the analysis and synthesis of 
observational data into requirements documents should be accomplished by teams, 
since teams bring in multiple perspectives to minimize the influence of individual 
biases. A typical team may consist of experts in a company or industry, social 
scientists experienced in describing and identifying social and cultural factors, 
designers, and perhaps the people who are expected to use the resultant 
technological system. 

The importance of supporting informal collaborative work have been demonstrated. 
In many cases the social interaction involved in building consensus is crucial. A 
team working on complex problems establishes a common context regarding the 
meaning of data by discussing, analyzing, and repeatedly reviewing it together. 

Researchers examining the use of technologies for group work point out the 
importance of natural interactions between team members, especially when using 
computers to assist or support the team's activities. Issues involve shared displays, 
turn-taking, recording activities and modes of interaction with objects. 

Design Elements 

Analysis Framework 

In at least one embodiment of the present invention, data collection methods are 
employed such as video ethnography and contextual inquiry to observe everyday 
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work practices, organizations' activities, and consumer behaviors. The practical 
benefits of these methods are well-documented. They ensure a rich understanding of 
an environment, situation, or activity, which is to be supported by a new or changing 
technology solution. 

5 

Analysis is typically done by looking for patterns in the data. Building on previous 
approaches, a simple analysis framework was created, which structures the data and 
allows for the use of numerous clustering and brainstorming techniques throughout 
the analysis, synthesis and design phases of a project. This simple structuring 
10 provides for a bottom-up approach for reaching the highest levels of data abstraction, 
going from data to requirements. At the same time, hypotheses are used for a top- 
down approach to ensure full coverage of a problem domain. 

Figure 8 diagrams the elements of the Analysis Framework, which are described 
15 below, while Figure 9 presents an example for each element. 

• Evidence is the basic unit of analysis. A piece of evidence is a significant action, 
quote, or description taken directly from the multimedia data gathered during 
observation. A project usually has hundreds to thousands of pieces of evidence, 

20 each of which is tagged to a piece of media. 

• Keywords are important concepts that are used to assemble the evidence into 
meaningful subsets during the initial stages of analysis. A piece of evidence is 
usually tagged to multiple keywords. In the past, a project may have fifty 
keywords. 

25 • Patterns are meaningful groupings of evidence derived from a bottom-up 
clustering of evidence. The projects typically result in fifty to one hundred 
patterns. 

• Hypothesis are a special type of Pattern. They are key questions the team needs 
to address when gathering data. Evidence is gathered to confirm or disconfirm 

30 hypotheses. 
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• Themes are the important dimensions of a problem domain that emerge during 
the analysis process. 

Experience shows that every project would have varying needs with regard to the 
data to be captured and analyzed. The analysis framework is simple, yet flexible, 
providing the freedom to assign specific types of labels to data instead of creating an 
overly structured, and therefore inflexible analysis framework. For example, in a 
current project "suggested solutions" and "surprises" keywords were used to 
organize subsets of evidence that may not directly relate to the bottom-up approach 
of identifying themes and patterns from evidence. 

Following the approach of others who found that new tools need to work in concert 
with many other team activities the present invention allows a team to use any 
number of techniques or tools to transform evidence into patterns, and patterns into 
themes. For example, cluster analysis was useful in a project in which there was 
interest in the intersection of evidence across many keywords. Similarly, affinity 
diagramming is used to identify themes from patterns. 

Managing the Data 

The present invention utilizes software tools to digitize, store, index, and access 
large amounts of multimedia data including the video, audio, and photographic data 
gathered during data collection. All of the elements of the analysis framework just 
described are also captured and stored. 

MAST (Multimedia Analysis Software Tool) was developed for use throughout the 
entire analysis and synthesis process. MAST supports data capture and storage, data 
entry, and searching and display capabilities. Digitized media clips can be played 
and views of organized subsets of the local data can be displayed- All of the 
interactions between the objects, devices, equipment, and displays of the present 
invention are managed by MAST. 
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The best way to quickly and easily view data is to digitize it. Providing easy, non- 
linear access to the original source data is essential. For example, during analysis a 
team views media clips to create consensus about the meaning of pieces of evidence 
and to clarify their interpretations of events that transpire in a particular media 
segment. 

MAST, as shown in Figure 10, is the glue that holds all of the elements of the 
present invention together. In the lab, team members can work with Interaction 
Objects, use the Interaction Devices to input, save, and retrieve data, view materials 
on the Display surfaces, and write on the whiteboards. The use of these objects and 
devices are detailed hereinafter. 

MAST maintains the links between multimedia data and evidence, and links 
between the various analysis elements as well as storing electronic whiteboard 
drawings. Providing this automatic "traceability" of the analysis and design process 
is a key feature of MAST. Preserving the history of the team's activities provides a 
rich design rationale. There have been numerous efforts to capture design histories 
often requiring the team to take extra steps to document their work. The approach to 
this challenging problem is to make it an easy, almost invisible part of the analysis 
process. 

The value of the captured history is apparent during the analysis process. It allows 
teams to easily review their work when there are questions about how they arrived at 
particular conclusions. In creating the final project documents accessing the captured 
history provides linkages between the underlying empirical data and the findings and 
issues which shape the final outcomes. 

Displaying the Data 

The ability to manipulate and view large amounts of data at one time is critical 
throughout data analysis. The present invention supports the use of a multi-monitor 
computer display, as well as using walls as displays. 
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Three 800x600 resolution displays provide a large surface for groups to view the 
MAST information and see several levels of data detail simultaneously. The lab of 
the present invention is long and narrow, 9 feet by 27 feet space prevented one from 
5 using three contiguously displayed high-resolution projectors on the long wall 
because a projector's focal length is more than nine feet. Therefore, the displays 
were placed on the nine-foot wall using a projector in the center and thirty-one-inch 
color monitors on the left and right, as shown in Figure 11. The different sizes of the 
displays creates a discontinuity causing some problems when moving the mouse 
p* 10 between screens. 



> I The different sizes of the displays also presents a challenge regarding readability. It 

m was determined that the content of each display should be legible from twelve feet 

^ away, since teams sitting around a table are likely to be seated at that distance. To 

M 15 achieve the perception that text displayed on each screen is approximately the same 

size, the smaller monitors use a larger font size than the center projector. A lOpt. 

font is used on the center display and an 18pt. font on the two side monitors. As can 
ill be seen from the picture in Figure 11, the relative sizes of the fonts in the windows 

appear comparable. The use of varying font sizes constrains the movement of 
20 windows from one monitor to the other. 

The MAST application consists of a combination of interaction windows and 
windows that are used for display only. Windows that require user interaction are 
placed on the center screen, whereas video clips are played on the left monitor, and 
25 additional details of data is viewed on the right monitor. The interaction windows on 
the center screen are used to create and maintain links among different levels of data. 
Example windows are shown below in Figures 12 and 13. It should be noted that the 
windows may depict the remote data in the form of video, images or the like. 



30 



During the knowledge management project all of the lab walls were covered with 
diagrams, notes, matrices, drawings, and affinity diagrams. It was found very useful 
to have easy access to the materials. This was important for comparing different 
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ideas at one time, as well as quickly and easily moving items around to make new 
groupings of information. Even the use of three high-resolution displays does not 
provide sufficient display space so there is still reliance on walls to provide 
additional display surface. Thus, in designing the present invention it was desired to 
5 preserve this capability, while also creating connections between the papers and 
sketches with the materials stored in the computer. When the walls are covered with 
paper linked to the computer via barcodes, the entire room, in essence, becomes a 
huge computer display. Note Figure 14. 

3 10 Connecting Paper to the Computer 

1 

\ The uses of paper and whiteboards by teams is well understood. Paper is used for 

reading, sketching, and annotating. Similarly, whiteboards are used for sketching, 
capturing emerging thoughts and viewing materials. The use of whiteboards in the 
15 present invention is discussed hereinafter. Research shows how individuals annotate 
paper to gain an understanding of what they are reading. Paper is also used to 
spatially display a lot of material at one time, and to easily move between documents 
or sections of documents. 



20 The key to the success of the present invention is the creation of a seamless 
interaction between the computer and these other forms, namely paper and 
whiteboard. Utilized were a barcode reader, a custom designed barcode command 
slate, and barcode enabled paper forms, which is called "Linked Sticky Notes" and 
"Linked Data Reports," to facilitate this interaction between paper objects outside of 

25 the computer and the data stored in the computer. 

Just as Video Mosaic connects paper-based storyboard elements, digital elements 
and video clips through the use of scanners and barcodes, the basic components of a 
design project were connected using barcodes. DigitalDesk supports activities that 
30 utilize pre-existing paper documents. For example, the calculator tool retrieves 
numbers from a printed page using image processing technology eliminating the 
need to punch the numbers into a calculator before they can be manipulated. These 
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two systems provide important evidence of the ways in which paper usage is an 
integral part of many activities. 

In accounting for the use of the paper, sticky notes, and whiteboards, a concerted 
5 effort was made to understand where a team's focus is at any given point in the 
analysis process and what activity they are doing. When team members are working 
individually, there is support for the use of personalized printouts. When they are 
discussing items there is support the use of the personal printouts, as well as the 
u computer displays and whiteboards to create a shared workspace. The following 
_2( 10 sections provide examples of how the team uses paper during different points in the 
P analysis process. 

'""::„ % 

MAST can generate any number of customized data reports filtered or sorted using 
various criteria. The example below shows a report of evidence sorted by keywords, 
r l 15 Associated with every item on a report is a barcode identifier that can be scanned to 
P display details about the item. Scanning the barcode for a piece of evidence plays its 

"si; 

p digitized video or audio clip. 

Ljl 

The first step of data analysis is to create patterns from evidence. The team begins by 
20 reviewing all of the evidence using linked evidence reports, or lists, as shown in Figure 
4. The team members spend time individually reviewing the evidence and making 
annotations on their own printed reports, which is followed by a discussion about the 
pieces of evidence. When hundreds to thousands of pieces of evidence are collected by 
multiple people, individuals may need additional information to understand evidence 
25 they have not seen before. To gain context, they can view a corresponding video clip for 
any piece of evidence by scanning the barcode associated with a piece of evidence listed 
on a report. Scanning a barcode to review a media clip is simpler and faster than using 
the computer to search for the particular clip to play. Similarly, when the team is already 
using the printed reports, it is far simpler to scan barcodes to link a piece of evidence to 
30 a pattern than it is to turn attention to the computer to open windows, search for an item 
in a window and drag-and-drop it to a pattern window. 
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As a team conducts analysis and creates patterns and themes, MAST is used to 
create associations of evidence into patterns and themes. New patterns or themes are 
printed on 2"x3" magnetic cards, as shown below. A "Linked Sticky Note" can be 
placed on a magnetic whiteboard. 

5 

Scanning the barcode identifier on a pattern's Linked Sticky Note displays its 
underlying information on the computer monitor. There is a natural fluidity in the 
way that one can point to an object and scan it to view its underlying information, 
rather than turning attention from the object to the computer to search for the item. 
10 This retains the person' s focus on the object they are already using. 

Enabling a team member to easily trace the history of the pattern by viewing all of 
the underlying data ensures that the team is referencing the actual data, not working 
from memory. It is common for team members to review all of the evidence linked 
15 to a pattern; it helps the team maintain a common understanding of the meaning of 
the pattern. 

When the team is developing conclusions, extensive discussions are captured using 
the electronic whiteboard. Team members sketch ideas, diagram relationships 

20 between interesting elements of the domain, and articulate key points. As they are 
working at the whiteboard the team does not want to think about which drawings 
will be important. Thus, every image is automatically saved and stored. All of the 
electronic whiteboard images created by the team are digitally stored in the MAST 
database. These images, tagged with barcode identifiers, are printed and displayed 

25 on a tackable fabric wall. These are called "Linked Images". It should be noted that 
numerous Linked Images may be created using the electronic whiteboard. 

As themes begin emerging, Linked Images of previous analysis sessions are moved 
around the room and placed with images of related discussions. These images are 
30 easily associated with themes. They can be displayed on the large projection screen 
for easy viewing by the entire team, and can be retrieved for editing using Adobe 
Photoshop™ or other image editing programs. 
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The Barcode Command Slate provides shortcuts to initiate the actions most 
commonly used during analysis. It is a printed list of barcoded commands for 
invoking specific actions. The use of the Slate enables the team to maintain focus on 
5 their work by providing a better mapping between what they want to do and how 
they start it. 

To "Create a Pattern" they scan the "Create a Pattern" barcode rather than using a 
series of mouse moves and mouse clicks to find and open a series of windows that 

D 10 are needed to create patterns. The Barcode Command Slate has three different types 

O of commands which are shown in Figure 15. 

y • The Barcode Command Slate works in conjunction with the MAST interface. When 

Lib; 

a command is invoked the appropriate windows are opened and displayed, providing 
15 feedback that the Barcode Command Slate actions are working properly. For 
O example, when the "Create a Pattern" barcode is scanned, the "Action Pattern" 

si 

r window is opened and displayed. As pieces of evidence are scanned to add them to 

the pattern, the evidence is displayed in the Action Pattern Window. 

20 The use of the Command Slate and wireless keyboard extends the reach of the 
computer application, saves time, and makes complex actions simple. The 
Command Slate is particularly valuable when creating patterns from evidence 
reports and creating themes from pattern Linked Sticky Notes. 

25 The interplay between paper, sticky notes, and computer occurs throughout the 
analysis process. Using the Linked Interaction Objects together - Linked Sticky 
Notes, Linked Data Reports, and Linked Images - dramatically enhances the analysis 
process. The team can focus on the content of their work because the use of these 
objects minimizes the amount of effort required to access, manipulate, view, and 

30 create data. 
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The challenging part of analysis is developing the higher-level frameworks that 
explain or give meaning to a particular problem. A team arrives at these frameworks 
by repeatedly organizing and reorganizing the available data. People attach meaning 
to spatial relationships they create between the objects and materials they display. 
5 This improves how they interpret and remember the data. The use of Linked Sticky 
Notes allow the team members to easily move the content around on the whiteboard. 
For example, they can group the Notes into an affinity diagram. They can create 
themes, which associate all of the Linked Sticky Notes included in the diagram. The 
group can then either retain the diagram or move the Sticky Notes knowing that they 
10 have recorded the theme, which is also represented by a Linked Sticky Note. 



While there is continued development and enhancement of the present invention, 
y, much is learned about its strengths and limitations by using the lab. Recently 

15 completed is a project to describe information needs surrounding the use of a car; 
this will be used to identify new services that could be delivered to people with in- 
car computing and GPS (Global Positioning System.) Three researchers used the 
present invention for over a month to analyze behavioral data collected through 
numerous observational techniques, resulting in activity logs, video, audio, and 
20 photographic data. 



The experience reinforced the design decisions that were made with regard to team 
attention and focus. The team spent numerous days reviewing the evidence reports to 
create patterns. Using the present invention has dramatically improved the speed and 
25 effectiveness of creating these patterns. It was also found that all team members 
would only direct their attention to the computer displays as needed, such as during 
discussions about precise wording of a pattern. During later stages of analysis, the 
Linked Sticky Notes were effective in allowing the team to fluidly move back-and- 
forth between high-level issues and underlying data in a unified manner. 

30 

Alternate Embodiments 
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Some additional capabilities would enhance the use of the lab. For example, the 
Linked Sticky Notes were placed on the whiteboard to manually carry out affinity 
diagramming. To create a theme and associate patterns and images with it, the team 
members must use the command slate, barcode scanner, and keyboard. Instead, it is 
5 desired to pursue the ideas presented in SILK for using screen (Whiteboard) gestures 
to make the association between patterns. More information regarding SILK may be 
found in a publication, "Interactive Sketching for the Early Stages of User Interface 
Design" by Landay, J. A., and Myers, B. A., which is incorporated herein by 
reference. By circling a group of clustered patterns and images, a theme would be 
10 created. The use of audio interface components is also considered to enhance how 
data can be manipulated by teams in a way that increases their focus on the data, not 
interaction with the computer. 

The use of the present invention is currently being expanded into other aspects of the 
15 design and development process. There is investigation on the use of the present 
invention for design teams who are reviewing the Requirements documents. As 
such, hybrid paper documents have been developed that take advantage of the 
present invention capabilities by utilizing the best of paper, the media server, three- 
screen display, and the barcode scanner. The document enables team members to 
20 create electronic annotations, bookmarks, and keywords, which link back to a paper- 
based document. 

Collection of Remote Data 

25 As mentioned earlier, remote data is collected from the field and may take on many 
forms such as video, still pictures or surveys. Various types of data collection 
systems have been available in the past for use in conjunction with panels of human 
respondents. Some of the uses for such systems are to test consumer response to a 
new or existing product, or to sample the buying habits of the panel of respondents. 

30 The respondents can be asked questions verbally, with or without the assistance of 
visual aids, or they can be asked to read through a booklet which directs the 
respondents through a series of questions to be answered. 
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The respondents can be given pieces of paper to write answers to the questions being 
asked of them, or they can be provided with some type of electronic device which 
they use to record their answers electrically. Electronic systems presently available 
5 are capable of receiving the respondents' answers and storing those answers in real 
time into a computer system. Such electronic systems are also capable of displaying 
those answers in real time on a video monitor. The video monitor can show the 
numbers of like answers in a bar chart format, for example, and/or can show the raw 
, data (in numerical form). One such system presently available is known by the name 

O 10 AcuPOLL, manufactured by AcuPOLL Research, Inc., and is used by market survey 

.MSB; 

p companies. 

ni 

CP The presently available AcuPOLL electronic data collection system uses enclosures 
which contain several numerical keys (such as "0" through "10") for the individual 
15 respondents to use in answering questions. Such keypad enclosures are "dumb" 

Q devices in that they do not present information to the user of the keypad (the 

£ respondent, or other person who may be testing or setting up the system). Such 

M keypads may have a light or 

two that provides some information. However, even with that light, the units are still 
20 not "intelligent" devices, providing only minimal information. In addition, each of 
these dumb keypads can only sense, and then transmit to a remote location, a single 
keystroke (0 through 10, inclusive) of information that represents the answer for any 
particular 
question. 

25 

The heart of a data collection system, such as AcuPOLLT, is a main computer which 
stores data received from the individual keypads. Since several keypads (up to at 
least one hundred units) can be used in a single data collection system, each 
individual keypad must have some type of identification means so that the main 
30 computer can determine which individual keypad is transmitting information at any 
given moment. With 
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such means, the main computer can keep track of what answers are being given by 
which respondent, thereby enabling the data collection system to categorize the 
answers by type of respondent (male or female, for example). Since the keypads of 
the AcuPOLL system are "dumb" devices, they cannot be programmed to change 
5 their operating characteristics, and they also must have their identification means 
"hard" coded. In other words, each of the AcuPOLL keypads must have some type of 
identification means in hardware (rather than in software or firmware). Such 
identification means could be a programmable switch (such as a seven-position DIP- 
switch) within each keypad, or it could be merely by the use of connecting the 
10 correct wires from the main computer into the respective keypads (and being truly 
"hard"-coded). 

Consequently, the data collection systems which are available at the present time 
must be set up by a rather regimented routine so that the correct keypads (having the 

15 correct identification number) are placed at their proper locations. Each keypad must 
be either attached to the correct wiring, or each keypad must have its identification 
switch (if one exists) set to the correct number. All this must be accomplished while 
the set-up procedure takes place, in view of the fact that the physical location of each 
keypad (according to its identification number) is usually very important to the 

20 overall survey results. Such procedures can be very time consuming when setting up 
a new testing room. 

Data collection systems available at the present time also are limited in that the 
keypads cannot be used to send information to the person located at the keypads. 
25 With more sophisticated test sampling occurring in the future, it would be desirable 
to be able to communicate information from the main computer to each of the 
respondents who are using those keypads. Such messages could inform a respondent 
that the last answer given was not within the correct range of acceptable numbers, 
for example, for questions asking for only certain numbers as answers. 

30 

Presently available data collection systems are not capable of accepting several 
keystrokes from the keypads as answers to questions. In such systems, the 



-37- 



respondents are restricted to using only one keystroke per answer, thereby allowing 
for only a range of zero (0) through ten (10), inclusive. A data collection system 
having "smart" keypads, which have more capabilities are required for such 
increased capabilities. 

5 

Yet another means of collecting the remote data entails the use of cameras. Modern 
digital cameras typically include an imaging device which is controlled by a 
computer system. The computer system accesses raw image data captured by the 
y , imaging device and then processes and compresses the data before storing the 

y 10 compressed data into an internal memory. Efficient operation of the computer is 

ip therefore an important consideration for camera designers and manufacturers. The 

Hi 

■f- 1 memory architecture of a particular computer system determines data storage 

0 1 techniques and can thus significantly effect the operational efficiency of the entire 

a digital camera system. 

ft 15 

3 The user interface in conventional digital cameras typically includes a view finder in 

□ which small versions of the captured images may be displayed to allow a user to 

review several images at once. By selecting one of the images, the user may then 
display the full-sized version of the images in the view finder. 

20 

The images captured by the digital camera are typically named by numbering the 
images starting with an initial number, such as the number one, and ending with a 
maximum number. The images displayed in the view finder may then be identified 
by their assigned numbers. Once the maximum number of images has been named, 
25 the user must delete the images or download the images to a host computer in order 
to clear the camera's memory. Thereafter, the camera begins naming new images 
starting with the number one again. 

This conventional method for naming images results in several naming conflicts that 
30 may result in some images being overwritten and lost. One type of naming conflict 
occurs in cameras that are equipped with some type of removable storage medium 
for storing images. For instance, assume that the removable memory, such as a flash 
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disk, includes a series of old images that are named from one to the maximum 
number as described above. If the user begins capturing images, the camera will 
begin naming the new images starting with the number one since the maximum 
number has been reached. This results in the old images stored on the removable 
5 memory being overwritten by the new images that have the same names. 

Another type of naming conflict occurs when conventional cameras are connected to 
a host computer for transferring images. Assuming that both the host computer and 

M* the camera include stored images, then both the host computer and the camera may 

10 have a set of images with the same names. If any attempt is made to the download 

Q images from the host computer to the camera, then the images with the conflicting 

fll names in the camera may be overwritten. Similarly, if any attempt is made to upload 

ffjj 

images from the camera to the host computer, then the images in the host computer 
? with the conflicting names may be overwritten. 

£ 15 

7: In one method, the digital camera maintains a count of the total number of captured 

3 images stored in the camera, and then incorporates this number in the name assigned 

to each image. Although this creates unique image names for the images captured by 

a particular camera because the nonrepeating count total, naming conflicts may still 
20 arise when a single removable memory device is used to store images from a number 

of different cameras, because each camera will eventually generate the same 

images names. 

A digital camera typically includes an image sensor having an array of picture 
25 elements, generally referred to in the art as pixels, that are positioned contiguous to 
each other. Each pixel functions to form one discrete element of the electronic 
representation of the image which, when assembled together in the array, forms the 
total electronic image 
representation. 

30 

The image sensor operates by converting optical energy it receives into separate 
charge packets at each pixel. The amount of charge that is produced is dependent 
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upon the light level and time of exposure. The charge packets are then read out from 
each pixel and stored to a storage medium, for example a well known PCMCIA 
ATA memory card (memory card), or internal memory of the digital camera for later 
retrieval. 

5 

The stored images may then be exported to a personal computer (PC) system for 
display on its PC monitor, such as for example a typical cathode-ray-tube screen 
(CRT), or for printing a hardcopy via a printer. For such exporting, the memory card 
}!: may be inserted into a PCMCIA drive of the personal computer for permitting user 

O 10 interaction. Alternatively, the content of the memory card or internal memory may 
rj be downloaded to the personal computer through a serial communication cable by 

^ the image manipulation software program. Presently, this is the only method of 

M< retrieving images stored in the internal memory. 

15 The images are temporarily stored in random access memory (RAM) by such image 
y manipulation programs during its operation. However, it may be desirable to 

permanently store the images on either a hard disk of the computer, or a computer- 
insertable floppy disk. In the case of permanent storage, an operating system (OS) 
that resides on the computer will create an entry on a disk file directory and then 

20 update the file allocation table, FAT. The file directory contains all basic file 

information such as file name, date and time of creation and location of FAT. The 
FAT is an indexing system for the OS to indicate the location of each file on the 
selected disk. Both the file directory and FAT are permanently stored on the 
particular disk selected, typically a floppy or hard disk. 



25 



30 



The file directory and FAT comprise a standard file system used by computers 
utilizing a "WINDOWS" or DOS-based OS for indexing each particular data file to a 
storage location, and in the present case, it indexes the location of each data file 
representing a recorded image. 

Although the presently known and utilized system and method for retrieving images 
from a digital camera are satisfactory, they are not without shortcomings. First and 



-40- 



foremost, most desk-top personal computers do not have a PCMCIA drive, and in 
this case, access through the serial port is the only means of retrieving images. In 
either case, the recorded images are received by the image manipulation software 
program in compressed form from the camera, and are de-compressed by the 
5 software for permitting either display, user manipulation or printing. For 
permanently storing the images, the images are either compressed again for 
minimizing memory storage size or stored in decompressed form. However, this 
recompression, if it uses a well known lossy compression method for obtaining a 
higher compression ratio, can cause further loss of detail, and if the image is not re- 
D 10 compressed, a large portion of memory space is obviously consumed for storage. 



J/f Retrieval of Data 

When faced with ill-defined problems requiring information access, one often wants 
to explore the resources available before exploiting them. This exploration may be 
f* 15 partly aimed at refining the understanding of the potential space of content that is 
O available, and partly aimed at formulating a concrete course of action for getting 

4 specific documents. Interfaces that support the browsing of a collection, as opposed 
H to searching a collection, are aimed at satisfying this need to learn more about a 

collection before taking action. 

20 

Scatter/Gather is a document browsing technique aimed at supporting such 
exploratory learning. The emphasis in this browsing technique is to present users 
with an automatically computed overview of the contents of a document collection, 
and to provide a method for navigating through this summary at different levels of 
25 granularity. The central primitive operation in Scatter/Gather involves document 
clustering based on pairwise document similarity. The technique aims to place 
similar documents into the same cluster. Recursively clustering a collection produces 
% cluster hierarchy. 

30 For each cluster, at each level of this hierarchy, the user is presented with summary 
information about the cluster that presumably communicates something about the 
kinds of documents it contains. The user may then select (gather) those clusters that 
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seem interesting or relevant. This subset of clusters can then be reclustered 
{scattered ) to reveal more fine-grained clusters of documents. With each successive 
iteration of scattering and gathering clusters, the clusters become smaller and more 
detailed, eventually bottoming out at the level of individual documents. 

5 

Bar Code Readers and Hyperlink methods 

Various types of hand-held bar code readers such as wands, fixed beam, moving 
beam, and linear CCD readers, have been developed for many applications. Wands, 
M fixed beam, and moving beam readers all operate using the same basic principle, 
if 10 Typically, a small spot of light is directed from a source of illumination to the 
CI surface of the bar code and is swept across the entire length of the bar code (i.e. a 
ih ; scan line). The intensity of the reflected light from the beam is modulated by the bar 

code pattern. This reflected light is gathered by optics and focused on an optical 
§ detector which converts light intensity to an electrical signal. Since light areas reflect 

•Li. 

■ Lh 15 more than dark areas, the reflected light intensity represents the digital code which 
serves to identify the content of the bar code symbol. The wand is the least 

p complicated of these readers and consists of a light source (typically an LED) and a 

photosensitive element in a pen-like package. The operator must sweep the wand 
across the length of the bar code while keeping the tip of the wand in contact with 
20 the bar code. This severely restricts the use of wands to applications where the 
operator can physically contact the bar code. Additionally, repeated sweeps of the 
bar code are typically necessary to obtain a valid read, increasing the overall time 
required to read each bar code. Wands will not read damaged or poorly printed bar 
codes. 

25 

A fixed beam reader operates with the same basic principles as the wand without the 
need to maintain physical contact with the bar code. The spot, typically produced by 
an LED or laser, is projected from the reader to distances up to about one foot. The 
operator is required to sweep the projected spot across the bar code. Fixed beam 
30 readers require very careful and accurate aim. Additionally, hand motions or jitter is 
amplified as the distance to the bar code increases. The longer the range, the larger 
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the bar codes must be. Fixed beam readers will not read damaged or poorly printed 
bar codes. 

Moving beam readers direct an LED or laser beam in a repetitive linear (i.e. one- 
5 dimensional) scanning pattern using rotating or vibrating mirrors and lenses. These 
readers eliminate the need for manual scanning or sweeping of the code. A moving 
beam reader typically scans at about 40 scans per second, allowing multiple tries on 
a bar code. This provides a better chance to get a valid read on a bar code with 
minimal damage. However the readers must get a complete valid scan (i.e. cut 
10 through all of the bars in one sweep) in order to decode a bar code. Moving beam 
readers are also restricted to reading small codes up close and large codes far away. 
Typical range of moving beam readers is about one foot. Moving beam readers will 
not read damaged or poorly printed bar codes. 

15 Linear CCD readers eliminate the need for mechanically sweeping a beam across bar 
codes, but require a bright source of illumination. Linear CCD readers capture a one 
dimensional image of the bar code and read out the information as an electrical 
signal which is similar to the output of wand, fixed, or moving beam readers. Linear 
CCD readers have an extremely small depth of field of about one inch, and are 

20 usually limited to reading bar codes shorter than the width of the CCD reader head 
(typically less than 3 inches). Linear CCD readers will not read damaged or poorly 
printed bar codes. 

CCD readers typically use light emitting diodes (LED) for illumination. Attempts 
25 have been made to improve the performance of linear CCD readers by using flash 
illumination. Examples of such readers are shown in U.S. Pat. Nos. 4,282,425; 
4,766,300; and 4,570,057. 

Electronic Drawing Boards 

30 It is conventional in a meeting that the attendants discuss a preset issue on the basis 
of papers prepared in advance and/or information written or drawn on a blackboard 
or whiteboard on site, that they check the contents of the issue and amend them as 
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needed, that the conclusion or interim results are recorded, and that a copy of the 
results of the meeting are distributed to the attendants and persons concerned. 

Incidentally, throughout the specification, the terms written and drawn, in all forms, 
5 should be considered interchangeable with respect to the act of placing, or existence 
of, information such as characters, commands, images, graphics, pictures, etc. on a 
blackboard or whiteboard, whether electronic or not. 

Electronic board systems are known as convenient means capable of facilitating a 
10 meeting. Such electronic board systems are disclosed, for example, in Japanese 
Patent Examined Publication (kokoku) No. 3-26855 and Japanese Patent 
Unexamined Publication (kokai) Nos. 2-200499, 1-125296, 63-209999, and 62- 
52630. 

1 5 The electronic board system as disclosed in Japanese Patent Examined Publication 
(kokoku) No. 3-26855 is configured in such a manner that characters, graphics, etc. 
written or drawn on a screen of the electronic board system, with and through a 
writing device with an oscillator equipped therein, are detected as a sequence of data 
of coordinate positions through a sensor that is embedded in the electronic board. 

20 The sequence of the coordinate position data is stored in image memory, and the 
traces of the characters, graphics, etc. indicated by the sequence of the coordinate 
position data are displayed as the characters, graphics, etc. on a display screen. The 
sequence of the coordinate position data existing in a given region is deleted from 
the corresponding image memory, with and through a deleting device with an 

25 oscillator equipped therein, by performing operations for deleting the displayed 
coordinate position data from the given region on a screen of the electronic board, 
and the characters, graphics, etc. displayed in the given region are thereby deleted 
from the display screen. 



30 



The electronic board system as disclosed in Japanese Patent Unexamined 
Publication (kokai) No. 2-200499 is so arranged as to recognize characters, graphics, 
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etc. written manually with a writing device on its board through a recognition 
processing device, and to record the result of recognition on recording paper. 

Further, the electronic board system as disclosed in Japanese Patent Unexamined 
5 Publication (kokai) No. 63-209999 is configured in such a fashion that characters 
written manually are read through an image reader and recognized with an OCR, and 
the result of recognition is displayed on a screen of the electronic board in an 
enlarged manner. 

p 10 In addition, the electronic board system as disclosed in Japanese Patent Unexamined 
CI Publication (kokai) No. 1-125296 is configured to recognize voices entered in a 

voice recognition device as letters, and to display the words corresponding to the 
u recognized voices on the screen of the electronic board. 

15 Furthermore, Japanese Patent Unexamined Publication (kokai) No. 62-52630 

discloses an electronic board system which is configured to recognize characters and 
- graphics entered manually on a transparent graphics tablet device through a stylus 

input device and to display the result of recognition on a display device, such as a 
liquid crystal screen, integral with the graphics tablet device. 

20 

In addition to those described hereinabove, a device is proposed in an article entitled 
"A New Face of User Interface: Features of Two Big OSs for Input through Pens" 
("Nikkei Byte" Magazine, April, 1991, pages 233 to 238), which can translate 
particular traces of a pen on a graphics tablet device integral with a liquid crystal 
25 screen into edit commands, and edit the corresponding image displayed on the liquid 
crystal screen. 

Furthermore, as introduced in the leaflet "Smart 2000 Series Products" of Smart 
Technology, Inc., Canada, a data processing system is proposed which allows the 
30 user to perform all operations in an interactive manner on a screen of the electronic 
board by recognizing characters, graphics, commands, etc. written through a 
pointing device on the screen of the electronic board, and projecting the result of 
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recognition or the result of execution of the commands through an LCD projector on 
the screen thereof and displaying it in a real time basis thereon. 

Although only a few embodiments of the present invention have been described in detail 
herein, it should be understood that the present invention may be embodied in many 
other specific forms without departing from the spirit or scope of the invention. 
Therefore, the present examples and embodiments are to be considered as illustrative 
and not restrictive, and the invention is not to be limited to the details given herein, but 
may be modified within the scope of the appended claims. 



